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Purpose  of  the  Study.  The  study  was  performed  to  assess  the  cost  benefit  of  a  Department  of  Defense 
(DoD)  Joint  Modeling  and  Simulation  System  (JMASS)  -  a  common  engineering  and  engagement  modeling 
and  simulation  (M&S)  system  to  support  acquisition. 

Background  of  the  Study.  The  Department  of  Defense  is  seeking  ways  to  improve  efficiencies  in  M&S 
by  creating  systems  that  foster  reuse,  interoperability,  and  commonality  among  model  components,  models, 
and  simulations.  The  Services  conducted  a  series  of  pilot  projects  using  Air  Force  (AF)  JMASS  in  order  to 
refine  the  requirements  in  the  DoD  Joint  Initial  Requirements  Document  (JIRD)  and  determine  if  a  joint 
M&S  system  will  support  acquisition.  It  was  anticipated  that  use  of  AF  JMASS  would  generate  benefits  in 
cost,  schedule,  quality,  and  productivity. 

Principal  Results.  The  recommendation  of  the  study  was  for  DoD  to  make  an  investment  in  the  AF 
JMASS  system  through  the  Program  Objective  Memorandum  (POM).  Funding  would  be  directed  to 
improve  existing  tools  and  services  of  the  system.  The  improvements  will  reduce  the  risk  in  the  acceptance 
of  the  system  by  the  analysis  community.  Rather  than  trying  to  cover  the  entire  acquisition  process  and  the 
huge  fiscal  burden  that  initial  model  development  would  require,  concentrate  on  populating  the  model 
repository  with  threat  and  environment  models  to  meet  all  the  Services’  requirements  in  the  domains  of 
aircraft,  missiles,  helicopters,  and  unmanned  aerial  vehicles.  To  minimize  cost,  leverage  AF  JMASS’ s 
established  models  and  system.  Other  reasons  for  making  an  investment  in  the  current  AF  JMASS  system 
follow:  it  is  technology  for  the  future  -  object  oriented;  it  supports  a  model  repository  approach;  it  fosters 
reuse  of  model  components  and  models  within  and  outside  programs;  it  eliminates  legacy  duplication  of 
effort;  it  is  a  better  product  than  legacies;  and  it  can  be  used  as  a  standard  for  industry. 

Time  of  Study.  The  JMASS  Senior  Steering  Group  (SSG)  decided  to  make  a  mid-year  POM  decision  on  4 
March  1998,  the  allocated  study  duration  then  became  15  September  1997  to  4  March  1998. 

Scope  of  the  Study.  The  study  was  limited  to  research  within  DoD  acquisition  programs  that  could  provide 
legacy  model  cost  estimate  data  across  a  common  domain.  The  common  domain  approach  provided  a  way 
to  find  common  legacy  model  use  examples  across  the  domain  by  a  number  of  programs,  illustrating  the 
duplication  in  use  of  legacies  and  illustrating  a  side-by-side  cost  comparison  of  legacy  use  to  a  comparable 
capability  in  DoD  JMASS.  The  cost  illustration  examples  were  expressed  in  “rough  order  of  magnitude” 
estimates. 

Study  Objective.  Examine  the  benefits  to  the  DoD  acquisition  community  from  having  a  common 
engineering/engagement  model  system  and  model  set  which  meets  all  the  DoD  JMASS  JIRD  requirements. 

Basic  Approach  of  the  Study.  The  study  was  accomplished  through  cooperation  and  interviews  with  the 
AF  JMASS  program  office,  current  users  of  AF  JMASS,  the  Service  pilot  projects,  Service  staff 
organizations,  acquisition  program  managers,  the  analysis  community,  and  private  industry  to  gain  as  much 
information  through  research  as  possible.  The  study  effort  culminated  in  a  briefing  with  insights,  findings 
and  recommendations  to  the  JMASS  SSG  to  assist  them  in  their  POM  decision.  This  report  provides 
documented  evidence  supporting  the  study. 

The  Study  Sponsor.  Office  of  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology 
(OUSD(A&T)),  Director,  Test,  Systems  Engineering,  and  Evaluation  (DTSE&E). 
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Main  Report 


Objective 


What  are  the  benefits  to  the  DoD  acquisition  community  of  having  a  common  engineering/engagement 
model  system  and  model  set  which  meets  all  the  DoD  JMASS  JIRD  requirements? 


Vision  for  the  Future 


Simulation  Based  Acquisition  (SBA)  is  the  robust,  collaborative  use  of  simulation  technology  that  is 
integrated  across  acquisition  phases  and  programs.  A  DoD  common  engineering  /engagement  model  system 
and  model  set  would  support  a  SBA  Process.  The  benefits  of  SBA  are  intuitive;  it  is  necessary  to  quantify 
them  in  order  to  clearly  determine  benefits.  The  DoD  common  engineering/engagement  model  system  and 
model  set  was  originally  envisioned  to  provide  a  standard  for  model  component  developers.  Tools  within 
the  system  would  support  model  development,  simulation  development  and  analysis.  There  is  a  need  to 
address  a  long  standing  question  on  the  quantifiable  and  qualifiable  benefits  of  a  common  modeling  and 
simulation  infrastructure  to  support  engineering  and  engagement  level  simulations.  It  is  desirable  to 
determine  the  benefits,  costs  and  risks  to  the  DoD  in  having  a  common  M&S  system. 


Background 


General.  OUSD(A&T)/DTSE&E  is  responsible  for  an  initiative  investigating  use  of  a  common  M&S 
system  within  the  DoD  acquisition  process.  A  JIRD  is  in  place  and  begins  to  define  such  a  system.  The 
Services  are  conducting  a  series  of  pilot  projects  using  AF  JMASS,  the  best  example  of  a  common  M&S 
architecture  for  acquisition,  in  order  to  refine  the  requirements  in  the  JIRD  and  determine  if  a  joint 
modeling  and  simulation  system  will  support  acquisition.  It  is  anticipated  that  use  of  AF  JMASS,  a 
common  M&S  system,  will  generate  benefits  in  cost,  schedule,  quality,  and  productivity.  These  benefits  are 
intuitive  but  not  documented.  There  is  a  need  to  document  these  benefits  for  DoD. 

Current  M&S  Methods .  Today  program  managers  use  threat  information  from  a  variety  of  sources  in  an 
amalgam  of  models  because  there  is  no  alternative.  Design  decisions,  tactics,  supportability,  and 
acquisition  issues  are  all  supported  in  part  by  a  set  of  models  which  may  not  be  valid,  which  may  not  be 
consistent,  and  which  do  not  interface  well  with  other  models. 

Recent  JMASS  work  by  the  U.S.  Air  Force.  AF  JMASS  is  an  Air  Force  directed  program  to  develop 
and  deliver  a  distributed,  object-oriented  architecture  and  system  focused  on  the  tactical  level  of  war 
(engineering  and  engagement-level  simulation).  As  a  system  for  developing  engineering  level  models 
primarily  of  electronic  systems,  AF  JMASS  will  provide  a  common  environment  for  tactical  modeling 
across  the  requirements  development,  acquisition,  and  test  process  spectrum.  The  Air  Force  is  currently 
working  with  the  Missile  and  Space  Intelligence  Center  (MSIC)  and  the  National  Ground  Intelligence 
Center  (NGIC)  to  populate  the  AF  JMASS  system  with  authoritative  representations  of  Air  Force  and 
threat  systems  that  interoperate  and  comply  with  AF  JMASS  standards.  A  study  of  the  AF  JMASS  Program 
was  recently  completed  by  the  Air  Force  Scientific  Advisory  Board.  This  study  endorsed  Air  Force  JMASS 
and  the  purpose  of  the  AF  JMASS  Pilot  Projects. 


Assumptions 
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•  The  study  will  quantify  the  benefits  to  DoD  if  a  common  modeling  and  simulation  system  which  meets 
all  the  JIRD  requirements  was  available  to  the  whole  acquisition  community. 

•  Authoritative,  quantifiable  information  about  acquisition  process  benefits  will  be  available. 

•  There  will  be  costs  incurred  with  the  development  of  this  system,  specifically  in  the  architecture  and 
conversion  of  existing  capabilities  and  program  databases. 


Limitations 


•  The  “quick  look”  portion  of  the  benefits  analysis  must  be  completed  in  two  months. 

•  Current  AF  JMASS  addresses  only  engineering  and  engagement  M&S  for  analysis. 


Methodolo 


Approach.  The  study  will  be  conducted  in  two  parts:  a  “quick  look"  assessment  and  a  “detailed 
assessment”.  During  the  conduct  of  the  study,  both  qualitative  and  quantitative  information  to  address  the 
study  objective  will  be  collected.  The  study  will  be  accomplished  through  cooperation  and  interviews  with 
the  AF  JMASS  program  office,  current  users  of  AF  JMASS  (e.g.  MSIC,  NGIC,  NAIC,  B-1B,  F-22),  service 
staff  organizations  and  private  industry  to  gain  as  much  information  through  research  as  possible.  The  final 
assessment  will  be  based  on  comparative  analysis  between  DoD  sources  and  private  industry’s  successful 
use  of  common  architecture  and  models  to  support  cost  savings.  The  private  industry  examples  provide 
solid  credence  to  any  assumptions  made  since  they  will  be  based  on  real  cost  savings  from  authoritative 
sources.  The  study  team  will  attend  project,  program  and  service  IPTs  to  gather  and  research  information 
needed  in  the  conduct  of  the  study. 

Metrics .  The  study  will  measure  the  benefits  of  an  M&S  system  as  it  relates  to  the  acquisition  process, 
specifically  what  savings  can  be  realized.  These  measures  are  documented  in  Appendix  H. 

•  What  is  the  cost  avoidance  from  eliminating  duplication  of  efforts  -  improved  collaboration  among 
programs  and  Services? 

•  What  are  the  cost  savings  across  the  Acquisition  cycle? 

•  Do  the  benefits  of  a  common  software  toolkit  reduce  redundant  investments  in  tools  and  services? 

•  What  are  the  front  end  costs  of  converting  existing  databases  to  a  new  architecture  and  set  of  models? 

•  What  are  the  benefits,  such  as:  schedule,  level  of  effort,  quality,  and  productivity  associated  with 
standard  model  architecture  use? 

•  What  are  the  benefits  of  reuse  and  interoperability? 

•  Can  alternative  design  concepts  be  readily  evaluated  on  a  common  "playing  field"  using  a  standard 
model  architecture? 

•  What  is  the  common  engineering  model  architecture  and  model  set  to  support  an  advanced  Simulation 
Based  Acquisition  Process  for  DoD? 

•  Are  there  quantifiable  benefits  of  improved  integration  across  the  DT  and  OT  communities,  using 
domain  validated  common  models  and  common  simulation  infrastructure? 

•  Does  a  standard  architecture  offer  potential  for  improved  requirements  definition  with  the  war  fighters? 
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What  are  the  benefits  of  improved  data  handling? 


Analysis 


General.  This  section  covers  the  analysis  performed  to  support  the  study.  The  basis  of  the  analysis  was  a 
data  collection  effort  consisting  of  interviews  with  industry  and  DoD.  The  interviews  allowed  the  study 
team  to  gain  quantitative,  qualitative,  and  insightful  information  to  answer  the  study  questions.  Interviews 
were  conducted  with  the  Air  Force  JMASS  Program  Office,  current  users  of  AF  JMASS,  the  Service  pilot 
projects,  Service  staff  organizations,  acquisition  program  managers,  the  analysis  community,  and  private 
industry  to  gain  as  much  information  through  research  as  possible.  The  analysis  section  is  divided  into  two 
parts.  Part  1,  the  Background  Analysis,  is  a  review  of  existing  information  and  an  analysis  of  DoD  and 
industry  M&S  benefits.  A  review  of  related  studies  provided  the  study  team  with  existing  information 
about  the  effectiveness  of  M&S  systems  and  the  existence  of  common  systems  within  the  Services.  A 
review  of  authoritative  references  allowed  the  team  to  put  in  perspective  where  DoD  JMASS  best  influences 
acquisition  and  whether  or  not  DoD  JMASS  might  be  a  single  system  or  a  system  of  systems.  A  look  at 
industry’s  decisions  and  investments  in  common  M&S  systems  allowed  the  team  to  understand  the  order  of 
magnitude  costs  industry  has  incurred  in  moving  to  common  systems,  what  prompted  their  decisions  to  go 
to  common  M&S  systems,  how  these  order  of  magnitude  costs  relate  to  a  common  M&S  system  to  support 
DoD  acquisition,  whether  or  not  industry  already  has  a  system  that  will  support  DoD,  and  whether  or  not 
there  is  some  existing  linkage  between  industry  and  DoD.  Part  2,  Cost  Benefit,  is  a  detailed  look  into  the 
existing  Air  Force  JMASS  development  community  and  provides  order  of  magnitude  cost  estimate  data  to 
illustrate  the  estimated  cost  and  the  benefits  of  the  system  through  reuse,  interoperability  and  commonality 
as  compared  to  the  current  use  of  legacy  systems.  The  cost  benefit  analysis  resulted  in  a  recommendation  to 
DoD  based  on  analytical  results,  insights,  and  findings.  Figure  1,  Study  Schematic,  shows  what  is  contained 
in  this  report. 
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Figure  1.  Study  Schematic 


Part  1  -  Background  Analysis 


Review  of  Related  Studies.  A  review  of  related  studies  provided  the  study  team  with  existing 
information  about:  the  effectiveness  of  M&S  systems;  recent  recommendations  from  the  Acquisition  Task 
Force  on  M&S;  the  existence  of  common  M&S  systems  within  the  Services;  a  framework  for  understanding 
the  major  Joint  simulation  systems;  rough  order  of  magnitude  cost  estimate  information  from  one  of  the 
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JMASS  pilot  projects;  and,  an  endorsement  for  the  use  of  JMASS  to  support  an  acquisition  program.  This 
information  was  gathered  from  the  following  studies: 

•  Christensen,  Harold  E.,  et.  al,  Technical  Report,  JMASS  EO/IR  Performance  Requirements  & 
Options  Study  in  Support  of  Aviation  Electronic  Countermeasure  Programs,  Part  I,  (Performance 
of  EO/IR  Simulation),  September  19,  1997,  and,  Part  II  (Implementation  of  Options  Study), 
October  21,  1997,  CAS  Inc.  Summary  -  JMASS  EO/IR  model  development  technical  approach  and 
rough  order  of  magnitude  cost  estimates  for  JMASS  model  development  for  the  EO/IR 
environment. 

•  Harshberger,  Edward,  R.,  Report  of  the  Simulation  Technical  Review  Panel  (Draft),  DRR-1539- 
OSD,  December  1996,  RAND.  Summary  -  A  technical  assessment  of  common  M&S  systems  that 
currently  exist:  Air  Force,  Joint  Modeling  and  Simulation  System  (JMASS);  Army,  TACTICS  -  A 
Common  Object  Request  Broker  Architecture  (CORBA)-Based  Distributed  Processing 
Infrastructure;  Navy,  Simulation  and  Modeling  of  Operational  Needs  (SAMSON). 

•  Parker,  Ted,  Vice  Admiral,  U.S.  Navy,  Retired,  Final  Report  of  the  Acquisition  Task  Force  on 
Modeling  and  Simulation,  June  10,  1994,  Chairman,  Acquisition  Task  Force  on  Modeling  and 
Simulation.  Summary  -  Recommended  that  DDR&E  work  with  the  Services  to  establish  a  Joint 
Program  Office  with  an  associated  Joint  Mission  Needs  Statement  to  formulate  a  M&S 
architecture  to  support  the  acquisition  process. 

•  Patenaude,  Anne,  et.  al.,  Study  on  the  Effectiveness  of  Modeling  and  Simulation  in  the  Weapons 
System  Acquisition  Process,  Science  Applications  International  Corporation,  October  1996. 
Summary  -  a  study  of  a  representative  number  of  acquisition  programs,  both  within  DoD  and  the 
commercial  sector,  provides  consistent  and  pervasive  evidence  that  M&S  used  effectively  provides 
substantial,  quantifiable  benefit 

•  Rothenberg,  Jeff,  A  Framework  for  Understanding  JWARS,  JSIMS,  and  JMASS  (Draft), 
December  1997,  RAND.  Summary  -  Analysis  of  the  three  systems  with  respect  to  their  potential 
to  satisfy  the  need  for  interoperability  among  DoD  models  and  simulations  and for  reuse  of  model 
components ,  data,  tools,  and  environments. 

•  United  States  Air  Force  Scientific  Advisory  Board,  Report  on  the  Joint  Modeling  and  Simulation 
System,  SAB-TR-97-02,  June  1997.  Summary  -  Found  JMASS  release  3.1  to  be  stable  and 
functional  and  viable  for  use  in  the  B-l  DSUP  Program  for  testing  -  an  endorsement  for  the  use  of 
Air  Force  JMASS. 

Review  of  Existing  Common  Modeling  Systems. 

General.  The  Report  of  the  Simulation  Technical  Review  Panel  (Draft),  DRR-1539-OSD,  December  1996, 
RAND,  cited  from  the  previous  paragraph,  highlighted  the  existence  of  three  common  M&S  systems  and 
provided  a  technical  assessment  of  the  systems:  Navy,  SAMSON;  Army,  TACTICS;  and,  Air  Force 
JMASS.  The  intent  of  this  section  is  to  provide  a  brief  description  of  each  system  and  review  the  technical 
assessment  observations  from  the  review  panel. 

•  Navy,  SAMSON  -  is  an  object-based  simulation  framework  that  provides  run-time  management  for 
a  distributed  group  of  software,  hardware,  and/or  human-in-the-loop  players.  The  system  was 
developed  to  satisfy  the  real-time  integrated  ground  testing  needs  of  the  avionics  and  electronic 
warfare  test  and  evaluation  (T&E)  community.  Sponsors  for  the  development  of  the  system  are  the 
Air  Force  Electronic  Combat  Integrated  Test  (ECIT)  Facility,  Edwards  AFB,  CA  and  the  Navy  Air 
Combat  Environment  Test  and  Evaluation  Facility  (ACETEF),  Patuxent  River,  MD.  Assessment  - 
Prototype  software,  not  implemented  for  multiple  users  or  multiple  facilities,  limited 
documentation  and  minimal  tools.  Development  tailored  to  near-term  needs  of  very  demanding 
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environment  -  HWIL  integration  test  facility,  large  numbers  of  legacy  hardware  and  software 
systems,  tight  timing  requirements.  Design  tradeoffs  made  for  speed,  not  generality.  Careful 
development  approach  coupled  with  good  prototyping  environment.  Focused  on  real-time 
requirements. 

•  Army,  TACTICS  -  is  a  set  of  interconnectivity  services  that  provides  the  application-oriented  user 
flexibility  and  ease-of-use  in  rapidly  configuring  simulations  composed  of  custom  developed  or 
legacy  code.  It  is  based  on  the  object  oriented  design  principles  and  takes  full  advantage  of  the 
benefits  offered  by  industry  object-oriented  standards  such  as  CORBA.  The  system  enables  the 
user  to  construct  process  flow  simulations  and  event-based  simulations  using  plug  &  play  objects 
that  interact  across  distributed  computing  platforms.  The  sponsor  for  the  development  is  the  U.S. 
Army  Tank  and  Automotive  Command  (TACOM).  Assessment  -  Immature  in  implementation  for 
development  and  combination  of  component  models  into  simulations .  Simulation  engines  not 

‘  wrung  ouf\  lacks  software  structural  model,  minimal  development  and  configuration  tools.  Most 
elegant  and  “ modern "  approach  from  a  computer  science  perspective  as  it  uses  CORBA  - 
implication  of  easier  interface  with  industrial  design  technology  an  important  enabler  for 
simulation-based  design.  Clearly  a  capable  system  not  fully  implemented. 

•  Air  Force  JMASS  -  provides  a  flexible  object-oriented  simulation  infrastructure  that  assists  model 
developers,  engineers,  and  analysts  in  the  development  of  digital  models,  configuration  and 
execution  of  simulations,  and  analysis  of  simulation  results.  AF  JMASS  supports  varying  degrees 
of  model  fidelity  from  the  less  complex  analytic  models  to  highly  emulative  models  which  mimic 
real  world  systems.  The  requirement  for  AF  JMASS  is  to  support  engineering  and  engagement 
level  analysis.  The  sponsor  for  the  development  is  the  U.S.  Air  Force,  Wright-Patterson  AFB,  OH. 
Assessment  -  More  mature  than  SAMSON  and  TACTICS  in  implementation.  Relies  on  a  uniquely 
developed  set  of  interface  standards  “CORBA-like”  APIs  and  port  objects,  developed  before 
CORBA  was  available.  Oriented  toward  model  and  simulation  development,  extensive  set  of  tools 
for  development,  configuration  and  analysis,  fully  defined  software  structural  model  and 
interfaces,  simulation  engine  and  environment  models  defined  and  implemented.  Most  capable 
system,  however,  there  is  some  concern  for  technological  obsolescence. 

Summary  of  Existing  Systems.  The  previous  reviews  and  assessments  of  the  existing  systems  provide 
documented  evidence  of  the  existence  of  the  systems  and  their  status.  It  was  not  the  intent  of  this  study  to 
draw  any  correlation  or  comparison  among  these  three  systems,  only  to  recognize  the  documented  strengths 
and  weaknesses  of  each  system  as  this  evidence  provides  some  concern  for  the  long  term  strategy  of  a  DoD 
JMASS  system  to  serve  the  broader  acquisition  community.  It  was  recognized  at  the  time  the  study  was 
initiated  that  the  most  mature  system  was  AF  JMASS  and  as  a  consequence  was  the  main  point  of  departure 
for  study.  This  was  especially  relevant  as  the  six  pilot  projects,  funded  by  the  Services  and  DoD,  would 
conduct  their  investigations  using  AF  JMASS  version  3.1,  the  most  current  version  at  study  initiation. 
Insights  gained  from  the  pilot  use  of  AF  JMASS  as  well  as  rough  order  of  magnitude  cost  estimate  data 
from  the  AF  JMASS  model  development  community  served  as  the  most  credible  data  available  for  the  time 
frame  of  the  study. 


Review  of  Authoritative  References. 

Definition  of  the  Acquisition  Process  by  DoDD  5000. 2 R.  DoD  Acquisition  Policy  5000. 2R  is  the  current 
authoritative  policy  document  for  acquisition,  which  establishes  a  management  framework  for  translating 
warfighter  mission  needs  into  mandatory  procedures  for  major  defense  acquisition  programs  and  major 
automated  information  system  acquisition  programs.  This  process  is  further  defined  by  a  logical  means  of 
progressing  through  specific  requirements/actions  (phases)  and  decision  points  (milestones)  to  ultimately 
field  an  operationally  effective,  suitable  and  survivable  system  for  Do D.  The  segments  of  the  process  are 
named  Determining  Mission  Needs  (MN),  Concept  Exploration  (CE)-phase  0,  Program  Definition  and  Risk 
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Reduction  (PDRR)-phase  I,  Engineering  and  Manufacturing  Development  (EMD)-phase  II,  and  Production, 
Fielding,  and  Operational  Support  (PFOS)-phase  III.  The  milestones  are  0,  I,  II  and  III,  located  between 
MN  and  CE,  CE  and  PDRR,  PDRR  and  EMD,  and  EMD  and  PFOS  respectively.  Figure  2  describes  the 
acquisition  process,  showing  the  five  segments,  and  the  actions  performed  across  each  segment 
(summarized  at  the  top  of  each  segment).  This  information  is  presented  as  a  quick  overview  of  the 
acquisition  process  and  provides  a  foundation  for  subsequent  analysis  of  the  JIRD. 


Figure  2.  DoD  5000.2  Acquisition  Process  Summary 

JMASS  JIRD  Analysis.  DoD  JMASS  as  defined  by  the  JIRD,  is  capable  of  addressing  most  of  the 
acquisition  process  except  Determination  of  Mission  Needs.  At  the  initiation  of  the  study,  the  JIRD  was  the 
only  authoritative  reference  available  that  described  the  overall  programmatic  requirements,  high  level 
capabilities,  and  operational  support  requirements  for  a  M&S  system  to  support  acquisition.  It  was  also 
understood  from  this  document  that  the  DoD  JMASS  Pilot  Projects  would  be  used  to  refine  and  further 
specify  the  requirements  for  a  robust  Joint  Operational  Requirements  Document  (JORD).  The  DoD  JMASS 
Pilot  Project  activities  occurred  simultaneous  to  the  execution  of  the  study.  Therefore,  the  JIRD  was  not  the 
final  requirements  document,  but  served  as  the  initial  requirements.  This  was  the  best  document  available 
at  study  initiation. 

DoD  JMASS  JIRD  Summary.  The  JIRD  specifies  that  DoD  JMASS  is  designed  to  support  acquisition  as 
defined  in  DoDD  5000.2-R.  DoD  JMASS  will  provide  tools  to  build  models  and  simulations.  Additionally 
it  will  foster  interoperability  and  reuse  of  models  from  the  subsystem/component  through  the 
system/engagement  level.  The  main  characteristic  of  the  system  is  to  provide  a  non-proprietary, 
evolutionary,  support  infrastructure  which  will  ensure  flexibility,  openness,  and  minimal  constraint  on  the 
users.  The  major  highlights  of  the  DoD  JMASS  JIRD  follow: 

•  Create  a  software  architecture  to  develop  and  configure  engineering/engagement  models, 
execute  simulations  and  post  process  data. 

•  Provide  tools  to  support  computer  aided  development,  configure  system  and  environment, 
execute  simulations  and  interface  to  legacy  tools. 

•  Establish  standards  that  include  guidelines  for  DoD  JMASS  compliant  models,  tools  to 
implement  the  compliant  models,  model-to-model  and  model-to-system,  tool-to-system 
interfaces,  guidelines  and  tools  for  porting  legacy  models  to  DoD  JMASS,  and  man-machine 
interface  modeling. 

•  Provide  a  M&S  reuse  library  of  DoD  JMASS  compliant  models. 
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•  Facilitate  interoperability  with  other  simulations. 

Functionality  of  DoD  JMASS  JIRD  to  DoD  Acquisition..  The  JIRD  specifies  that  DoD  JMASS  is  intended 
to  support  a  variety  of  analysis  activities.  These  analysis  activities  were  correlated  with  the  DoDD  5000.2- 
R  and  plotted  on  the  acquisition  process  to  indicate  where  they  logically  fit.  Figure  3.  Functionality  of  JIRD 
to  Acquisition  shows  this  correlation. 


Functionality  of  JRDIto  Axqpiattnm 


Figure  3.  Functionality  of  JIRD  to  Acquisition 

JMASS  JIRD  Functionality  Issues. 

The  JIRD  specifically  delineated  eight  functional  analysis  capabilities  required  of  DoD  JMASS:  1) 
operations  analysis,  2)  concept  formulation,  3)  engineering  discipline  specific  analysis,  4)  total  ownership 
cost  analysis,  5)  design  and  construction  analysis,  6)  test  planning  and  performance  prediction  analysis,  7) 
maintenance  and  operations  analysis,  and  8)  manufacturing  and  process  flow  analysis.  As  these  analysis 
activities  were  reviewed,  some  issues  as  to  the  JIRD  definition  of  what  DoD  JMASS  is  intended  to 
functionally  do  were  raised.  Figure  4.,  DoD  JMASS  JIRD  Functionality  Issues,  highlights  these 
functionality  issues.  It  is  known  from  the  JIRD  that  DoD  JMASS  is  intended  to  support 
engineering/engagement  modeling.  Acquisition  sources  note  that  some  engineering/engagement  modeling 
is  used  early  in  the  acquisition  process  to  support  Concept  Exploration  and  Operations  Analysis,  however  it 
is  not  used  exclusively.  Other  higher  level  M&S  tools  are  employed  -  campaign  and  mission  modeling. 
These  higher  level  tools  are  in  predominant  use  during  the  early  stages  of  the  acquisition  process.  As  was 
discovered,  the  JIRD  encompasses  more  functional  areas  than  engineering/engagement  models  can  support. 

The  triangle  overlaid  on  Figure  4  is  intended  to  show,  based  on  the  JIRD  functional  capabilities  as 
depicted  in  Figure  3,  the  amount  of  engineering/engagement  M&S  use  across  acquisition.  It  reflects  that 
there  is  little  use  of  engineering/engagement  early  in  CE  and  increasing  use  of  engineering/engagement  as  a 
program  moves  out  of  CE  and  into  PDRR  and  EMD.  This  increase  is  due  to  more  definition  of  the  weapon 
system  as  time  goes  on  and  the  increased  reliance  on  engineering/engagement  tools  to  address  analysis 
issues  for  the  program.  The  increased  fidelity  of  these  tools  depict  a  more  accurate  definition  of  the  weapon 
system  as  it  becomes  available,  which  attempts  to  further  reduce  risk  in  development  and  maintain  the 
desired  military  worth  for  the  customer.  Engineering/engagement  M&S  is  used  to  its  maximum  extent  in 
EMD  and  then  tapers  off  as  the  product/weapon  system  moves  to  production  and  fielding. 

Besides  Operations  Analysis  and  Concept  Exploration,  Total  Ownership  Cost  Analysis  and 
Maintenance  and  Operations  Analysis  are  not  engineering/engagement  M&S.  Cost  analysis  models  are 
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seen  as  separate  from  performance  modeling.  Cost  analysis  is  certainly  an  important  aspect  of  acquisition, 
but  it  is  a  requirement  for  the  whole  process  and  is  driven  by  a  completely  different  set  of  parameters. 
Likewise,  Maintenance  and  Operations  Analysis  is  not  engineering/engagement.  This  area  of  analysis  is 
viewed  as  being  a  part  of  supportability  analysis  in  the  fielding  and  operational  support  phase  of  acquisition. 
This  analysis  relates  more  closely  to  issues  of  logistics.  This  only  leaves  four  of  the  eight  functional 
capabilities  germane  to  DoD  JMASS,  and  they  are  predominantly  EMD  functions. 


JIRD  Functionality  Issues 


0  Concept  Formulation,  Operations  Analysis  -  This  is  not  exclusively  engineering/engagement  modeling. 
Campaign/Mission  models  like  THUNDER  etc.  are  primarily  used  here. 

A  Total  Ownership  Cost  Analysis  -  This  is  not  engineering/engagement  modeling.  Cost  analysis  models 
encompass  more  than  just  the  engineering/engagement  level. 

♦  Maintenance  and  Operations  Analysis.  This  is  not  engineering/engagement  modeling.  This  is  more 
specific  to  supportability  analysis. 


Figure  4.  DoD  JMASS  JIRD  Functionality  Issues 

JMASS  JIRD  Influence  on  Acquisition. 

The  JIRD  analysis  raised  two  questions: 

1.  Where  does  DoD  JMASS  best  influence  DoD  weapons  system  acquisition? 

2.  Does  the  JIRD  envision  DoD  JMASS  as  one  system,  or  a  number  of  systems? 

1.  Where  does  DoD  JMASS  best  influence  DoD  weapon  system  acquisition?  The  JIRD 
functional  capabilities  triangle  for  JMASS  clearly  places  its  greatest  emphasis  in  EMD  with  four  of  the 
eight  capabilities  in  that  phase  of  acquisition.  It  is  intuitive,  based  on  this  information,  that  DoD  JMASS  is 
to  have  its  greatest  influence  in  the  EMD  phase  of  acquisition  as  shown  in  Figure  5.  The  Army’s  Simulation 
Support  Plan  (SSP)  Guidelines  Engineering  Development  support  this  same  rationale  for 
engineering/engagement  M&S  as  depicted  by  its  functional  capabilities  triangle  superimposed  on  the  JIRD 
triangle  in  Figure  5. 
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DoD  JMASS  Influence 


Best  influence  isinEMD 


Figure  5.  DoD  JMASS  JIRD  Influence  on  Acquisition 


2.  Does  the  JIRD  envision  DoD  JMASS  as  one  system,  or  a  number  of  systems?  The 
acquisition  process  requires  low  fidelity  M&S  (campaign  and  mission)  and  high  fidelity  M&S 
(engineering/engagement)  to  cover  all  the  analysis  requirements.  As  has  already  been  determined  from  the 
only  authoritative  document  to  be  distributed  on  DoD  JMASS,  the  JIRD,  DoD  JMASS  is  specifically 
defined  as  an  engineering/engagement  M&S  system  only.  The  campaign  and  mission  M&S  requirements 
will  come  from  some  other  system  so  as  to  cover  all  aspects  of  M&S  analysis  for  the  acquisition  process. 
Figure  6,  DoD  JMASS,  a  System  of  Systems,  depicts  the  JIRD  with  its  functional  capabilities  imposed  on 
the  acquisition  process  against  the  bubbles  of  acquisition  analysis  requirements.  For  DoD  JMASS  to  cover 
the  entire  spectrum  of  acquisition  analysis,  its  general  description  of  operational  capability  would  have  to 
be  expanded  to  the  campaign  and  mission  level.  DoD  JMASS,  because  it  is  defined  with  a  narrow  scope, 
can  not  be  the  only  M&S  system  for  the  acquisition  process;  it  will  be  one  of  a  yet  to  be  determined  number 
of  systems  that  will  work  in  concert  to  cover  the  entire  acquisition  process. 

Insight  from  Data 

•  We  see  campaign/mission  M&S  used  early  &  late. 

•  Engineering/engagement  does  not  cover  the  whole  process. 


Need  a  System  of  Systems  to  Cover  the  Whole  Process. 


Figure  6.  DoD  JMASS  A  System  of  Systems 


Acquisition  Context .  Having  confirmed  DoD's  definition  of  the  acquisition  process  and  what  M&S  analysis 
it  requires  from  the  authoritative  sources  available,  the  larger  picture  to  be  determined  was  the  acquisition 
relationship  of  industry  and  DoD.  In  the  acquisition  process,  industry  and  DoD  are  concerned  with  two 
diverging  interests  as  each  progresses  to  a  final  product  for  consumer  use  (see  Figure  7).  DoD  is  primarily 
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concerned  with  military  worth  -  the  ability  of  the  product  to  increase  the  warfighting  capability  of  the  DoD 
organization  which  defined  the  need  for  the  product.  Industry  is  primarily  concerned  with  profit  -  the  sum 
remaining  after  all  costs,  direct  and  indirect,  are  deducted  from  the  income  of  producing  the  product. 
Military  worth  is  founded  upon  military  operational  context  -  the  ability  to  wage  war  successfully;  and 
industry  is  founded  upon  the  manufacturing  context  -  the  ability  to  make  a  quality  product  while 
maximizing  cost  efficiencies.  Both  DoD  and  industry  share  the  same  three  goals  in  respect  to  their 
products:  high  quality  (better),  made  in  the  shortest  amount  of  time  possible  (faster),  and  at  the  least 
amount  of  cost  (cheaper).  As  the  figure  illustrates,  both  DoD  and  industry  start  as  two  very  distinct  and 
separate  interests:  military  worth  vs.  profit;  yet  narrow  across  the  acquisition  process  as  the  two  interests 
merge.  This  merging  results  from  the  trade-offs  that  DoD  and  industry  are  faced  with  as  they  each  try  to 
maximize  the  three  distinct  goals  while  facing  fiscal  reality.  In  other  words,  the  added  performance  the 
warfighters  want  may  be  cost  prohibitive,  so  they  accept  a  little  less  performance  for  a  much  better 
product/weapons  system  price  from  the  manufacturers. 


Acquisition  Context 

(Diverging  Interests) 


DoD 


Military  Worth 
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Determirntton  of 

Progfim  • 

Knaineering  and 

Production, 
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Exploration 

Definition  and 

Manufacturing 

Fiolding /Deployment 

Risk  Reduction 

Development 

and  Operation*! 

;  : 

Support 

f 

j — 

Industry 


pop:  J MASS'*  *  as  defined  by  the  J1RD 


Milestone  0  Milestone  I  Milestone  II  Milestone  III 


Figure  7.  Acquisition  Context  -  Military  Worth  and  Profit 

M&S  Benefits  to  the  Acquisition  Process  .  The  acquisition  process  demands  analysis  throughout,  with 
manufacturing  emphasized  in  phases  II  and  III  (see  Figure  7).  Analysis  is  a  tremendous  part  of  the  early 
stages  of  the  acquisition  process,  especially  in  determining  military  worth.  M&S  analysis  will  shorten  cycle 
time,  run  more  iterations,  collect  more  data,  and  use  less  manpower  -  all  cost  efficiencies,  not  necessarily 
direct  cost  savings.  The  manufacturers  are  today  saving  money  (increasing  profit)  by  producing  products 
using  M&S  to  maintain  their  edge  in  the  marketplace.  Industry's  management  understood  that  M&S  would 
require  a  big  front  end  cost;  but  the  long  term  savings  and  ability  to  maintain  market  share  were  deemed 
worthy  of  the  investment.  Industry  successes  are  direct  savings  to  DoD  because  their  products  are  better, 
faster  and  cheaper  -  the  common  goals  of  both.  Concrete  examples  are  the  best  ways  to  justify  a  position 
and  make  a  point.  Figure  8  shows  data  points,  only  examples,  spread  across  the  acquisition  process 
illustrating  how  M&S  can  potentially  save  time  and  money  for  DoD  and  industry.  The  magnitude  of  these 
cost  savings  is  hard  to  quantify  because  they  are  based  on  potential  and  not  necessarily  concrete  figures. 
The  best  example  collected  of  a  single  system  and  process  across  manufacturing  which  assimilates  the  DoD 
JMASS  JIRD  requirements  of  interoperability  and  reuse  is  CATIA.  This  system  is  used  by  Boeing,  national, 
and  international  automotive  industries,  and  many  others.  The  data  points,  shown  in  Figure  8,  Example  data 
-  Benefits  of  M&S  Use  in  Acquisition,  come  from  a  variety  of  sources: 

•  Joint  Strike  Fighter  (JSF)  -  Joint  Analysis  Support  Team  (JAST)  Operations  Analysis  Study 
Plan  Apr  95.  M&S  to  a  virtual  manufacturing  environment. 

•  Non-Line  of  Sight  (NLOS)  Missile  “Modeling  and  Simulation”,  Sept  94.  Distributed  Virtual 
Simulation  on  a  common  terrain  database. 
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•  AEDC  Missiles  -  AEDC  Trip  Report,  Martha  Simmons,  ‘95.  Physics  based  simulation  tools 
anchored  to  a  high  quality  test  model. 

•  B2/USAF  -  Manufacturing  and  the  Nil  (National  Information  Infrastructure)  Report  94. 
Digital  data  with  a  common  data  management,  storage,  retrieval,  and  exchange  service. 

•  Boeing  -  PBS  Broadcast  “21st  Century  Jet,  777”  96-97  and  an  AW&ST  Report  94.  A  single 
set  of  M&S  tools  and  a  process. 


Examole  Data  Points  of  Usina  M&S 
(Order  of  Maanitude) 


DoD 


JSF 

NLOS  Missile 
AEDC  -  Missiles 

B2/USAF 

Boeing  (777)  & 
Chrysler  ( Intrepid) 


Industry 


Military  Worth 

► 


Profit 


Milestone  0  Milestone  I  Milestone  II  Milestone  III 


Figure  8.  Example  Data  -  Benefits  of  M&S  Use  in  Acquisition 

Industry  Investment  in  M&S.  Industry  is  embracing  the  M&S  revolution  as  a  part  of  its  acquisition 
reformation.  In  the  previous  examples  of  Figure  8,  potential  savings  were  noted  so  that  the  benefit  could 
be  depicted  for  industry  and  DoD.  This  allows  the  reader  to  see  that  DoD  is  moving  forward  in  this 
technology  arena.  DoD  program  managers  make  M&S  investments  when  and  where  it  makes  sense.  Their 
decisions  are  based  on  good  business  practice  -  if  it  achieves  the  goals  of  better,  faster  and  cheaper,  they 
will  invest  in  it.  Looking  at  specific  examples  in  industry  can  add  clarity  and  better  tell  the  M&S  story. 
Chrysler  and  Boeing  have  made  huge  investments  in  M&S.  They  invested  in  CATIA,  a  CAD/CAM  system 
in  use  by  over  9000  commercial  companies  across  the  world.  This  M&S  system  significantly  reduces  the 
need  for  prototypes  and  replaces  that  old  technology  with  computer  models.  Computer  models  allow  for 
reuse  within  a  program,  interoperability  between  new  product  lines,  and  the  elimination  of  duplication  of 
effort.  As  an  example  of  this  technology,  Chrysler  took  the  computer  model  door  from  the  line  of  one  type 
of  car  and  used  it  to  craft  the  shape  of  a  door  for  another  line.  Chrysler  has  used  M&S  to  reduce  testing  by 
50%.  Boeing  and  its  111  have  been  the  M&S  product  standard  bearer  for  major  industry.  Boeing  was 
faced  with  being  driven  from  the  aircraft  market  place  by  Airbus  if  they  could  not  find  a  better,  faster  and 
cheaper  way  of  doing  business.  Top  level  management  moved  to  Integrated  Product  Teams  and  M&S 
technology  to  regain  their  hold  on  the  market  place.  Figure  9  illustrates  the  investment  and  return  that 
CATIA  has  brought  to  Boeing.  It  required  a  huge  front  end  cost  ~$5B  and  training  for  10,000  employees 
for  6  months.  The  risk  was  that  239  aircraft  had  to  be  ordered  to  break  even;  orders  for  aircraft  have 
surpassed  that  mark. 
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Industry  Investment  Example 

Investment  In  CATIA 

Boeing  777  Hardware/Software  Investment  ~$5B 

Manpower 

10,000  employees 

Training 

6  months 

Benefits 

Findings 

•  Better  -  rework  reduced  27%, 
improved  assembly  tolerances  (0.5 
inches  to  0.023  inches). 

•  Faster- 17%  less  time  to  build  a 
future  component  through  reuse 
(767  strut) 

•  Cheaper  -  scrap  reduced  30% 

•  Establishment  of  single  design 
and  manufacturing  M&S  system 

•  Big  up-front  cost 

•  Break-even  at  the  239th  A/C 

•  See  no  link  between  industry 
CAD/CAM  and  DoD 

•  Industry  desires  to  connect  to  a 
DoD  M&S  system  without  cost 

Figure  9.  Boeing  CATIA  Investment  Example 

DoD  Investment  in  M&S .  Acquisition  reform  is  alive  and  well  in  DoD;  program  managers  are  making 
business  decisions  using  best  business  practices  so  that  their  product  will  be  better,  faster  and  cheaper  to  the 
DoD  customer.  Program  managers  are  leveraging  off  industry  and  business  standards  are  replacing 
government  standards.  This  will  in  fact  move  DoD  toward  a  better,  faster  and  cheaper  product,  but  it  does 
not  establish  any  M&S  standards.  It  allows  program  managers  to  move  forward  based  on  their  budgets  and 
visions,  but  industry  has  no  common  process  to  connect  directly  with  the  DoD  acquisition  M&S.  Since  the 
program  managers  of  the  over  200  major  programs  are  making  M&S  decisions  without  a  common  standard 
and  industry  is  the  manufacturer  of  the  product,  it  makes  good,  common  sense  that  a  standard  needs  to  be 
in  place  to  facilitate  DoD/industry  cooperation .  On  more  than  one  occasion,  industry  representatives,  many 
of  which  attended  the  DoD  JMASS  Industry  Days,  have  asked  that  DoD  establish  a  single  M&S  system 
with  which  they  can  easily  and  cheaply  connect  their  concept  models  to  DoD's  threats  and  environments  so 
that  every  industry  concept  model  would  get  a  fair  and  impartial  evaluation.  If  DoD  maintained  a  repository 
of  verified,  validated  and  accredited  threat  and  environment  models,  reuse  of  these  models  would  ensure 
consistency  within  a  program,  save  money  by  only  having  to  create  a  single  model,  and  allow  all  the 
programs  needing  the  same  models  to  use  them  (interoperability).  Industry  is  the  definitive  source  for 
manufacturing  and  the  M&S  technology  that  is  evolving  in  this  area;  DoD  is  the  definitive  source  for 
operational  analysis  and  should  focus  on  the  M&S  technology  needed  now  and  for  future  contingencies. 
The  establishment  of  a  single  system  for  DoD  acquisition  would  give  industry  a  standard  for  M&S 
interface,  promote  reuse  and  interoperability  among  the  DoD  acquisition  process  phases  and  programs,  and 
foster  the  seamless  transition  between  DoD  operational  analysis  and  industrial  manufacturing. 

DoD  M&S  System  Realities.  Reuse,  interoperability,  elimination  of  duplication  of  effort,  seamless 
transitions  between  industry  and  DoD,  and  better,  faster  and  cheaper  products  -  all  these  benefits  are 
possible  through  M&S.  Can  DoD  afford  the  cost  that  a  single  system  would  demand?  Chrysler  spent 
millions  and  Boeing  ' s  estimate  is  in  the  billions  for  the  CATIA  M&S  system.  They  are  manufacturers  and 
they  found  a  system  that  meets  their  needs.  The  return  on  investment  (ROI)  they  envisioned  could  be 
directly  linked  to  maintaining  market  share  and  profit  -  future  and  near  term  gain.  They  were  also  willing 
to  step  out  smartly  with  new  M&S  technology  and  scrap  all  their  old  M&S  systems.  DoD  is  a  different 
story.  DoD  is  faced  with  a  shrinking  budget  and  must  find  ways  to  economize.  Millions  of  dollars  have 
been  invested  in  legacy  M&S  systems,  government  analysts  have  used  them  for  years  and  are  resistant  to 
change  to  new  systems.  The  competition  for  the  shrinking  budget  is  very  keen  due  to  the  numerous 
priorities  vying  for  the  dollars  within  each  Service  in  DoD.  Many  government  officials  see  only  the 
additional  cost  that  a  new  M&S  system  demands,  and  are  not  willing  to  wait  for  the  ROI  that  will  come  with 
time.  They  also  would  like  to  see  legacy  systems  folded  into  the  new  system  so  that  prior  investments  in 
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M&S  can  continue  to  be  used.  Services  are  not  easily  swayed  to  invest  the  large  up  front  dollar  amounts 
required  for  a  DoD  wide  M&S  acquisition  system,  unless  they  individually  get  some  return  on  their 
investment.  The  only  way  that  can  happen  is  to  find  products/platforms  that  will  cross  Services’ 
boundaries. 

M&S  can  certainly  save  time  and  money,  but  M&S  carries  with  it  the  risk  of  technological  obsolescence. 
Translators  change  every  six  months,  hardware  every  year  and  one-half,  and  software  every  five  to  six 
years.  Yet,  the  longer  DoD  waits  to  find  the  "perfect,  single"  system,  the  more  out  of  touch  it  will  be  with 
the  advances  being  made  in  the  commercial  sector.  In  the  review  of  the  JIRD,  DoD  JMASS  is  not 
functionally  capable  of  covering  the  entire  acquisition  analysis  process.  The  JIRD  is  not  the  final 
requirements  document.  The  JORD  and  the  Mission  Needs  Statement  (MNS)  will  be  the  defining 
documents  for  DoD  JMASS.  It  is  apparent  that  DoD  acquisition  will  require  a  set  of  M&S  systems  to  cover 
the  whole  process. 

Analysis  Summary  Part  1.  How  does  DoD  move  into  the  M&S  future  with  the  constraints  that  are 
realities  of  today? 

Establish  a  Joint  Program  Office  (JPO)/Task  Force.  If  the  DoD  acquisition  community  is  going  to 
move  forward  with  a  M&S  technology  standard,  it  will  not  come  about  from  the  program  managers.  DoD 
management,  like  industry’s  management  has  already  done,  must  take  positive  steps  to  aggressively  seek 
out  and  discover  the  set  of  systems  that  will  support  all  the  facets  of  DoD  acquisition  and  interconnect  with 
commercial  industry.  The  establishment  of  a  JPO  will  keep  the  focus  on  a  DoD  acquisition  M&S  system 
that  can  come  to  fruition.  This  will  work  if  DoD  stakeholders/users  (Services,  OSD,  acquisition 
community,  analysis  community)  are  represented  within  the  JPO.  Timing  the  implementation  so  that  the 
users  want  the  new  M&S  because  it  is  as  easy  to  use  as  the  legacy  systems  they  are  accustomed  to  and  it 
adds  capability  will  minimize  the  risk  of  investing  in  M&S  that  will  not  be  used  or  lacks  the  capability  to 
meet  the  goals  of  the  acquisition  community.  The  JPO  must  maintain  flexibility  to  incorporate  current 
technology.  A  methodology  must  be  designed  by  the  JPO  that  maximizes  benefit  across  the  acquisition 
process  and  Services,  yet  minimizes  expense  due  to  the  shrinking  DoD  budget  and  the  investment  already 
made  in  legacy  M&S  systems. 


Part  2  -  Cost  Benefit 


Time  Limitation.  Due  to  the  limited  duration  of  the  study,  the  aircraft  related  survivability  analysis  domain 
was  identified  as  the  sample  domain  to  examine. 

Cost  Benefit  of  DoD  JMASS . 

This  portion  of  the  study  will  document  “order  of  magnitude”  cost  estimates  of  DoD  JMASS  as 
applied  in  a  common  weapon  system  domain  and  the  benefits  of  the  system.  A  series  of  cost  illustration 
examples  will  be  presented  in  order  to  show: 

•  the  cost  of  DoD  JMASS  models 

•  the  cost  of  DoD  JMASS  models  compared  to  legacy  models 

•  the  cost  and  benefit  of  reuse  of  DoD  JMASS  models  within  a  common  domain 

•  the  cost  and  benefit  of  reuse  of  DoD  JMASS  models  across  multiple  domains 

The  cost  estimates  are  related  in  a  real-world  setting.  In  other  words,  they  show  the  potential 
benefit  to  acquisition  by  showing  cost  examples  within  a  setting  of  today’s  program  status.  The  cost  benefit 
is  conveyed  with  consideration  of  the  timing  and  implementation  of  DoD  JMASS  in  the  current  acquisition 
picture.  Note  however,  the  DoD  JMASS  system  itself,  as  represented  in  the  upcoming  cost  examples,  is 
strictly  a  notional  representation  of  the  system  and  is  yet  to  be  defined.  The  notional  representation  of  DoD 
JMASS  should  consist  of  the  three  distinct  model  entities:  the  digital  system  model  (DSM)  or  the 
representation  of  the  DoD  weapons  platform,  the  threat  model,  and  the  environment  model.  All  three  of 
these  model  entities  must  be  built  and  made  available  to  interact  with  each  other  to  make  the  complete 
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simulation.  The  cost  examples  to  follow  will  use  this  notional  representation  as  the  criteria  for  establishing 
cost  profiles.  The  closest  object-oriented  simulation  system  that  afforded  concrete  cost  figures  was  AF 
JMASS  and  this  became  the  basis  of  DoD  JMASS  costs.  Also,  from  this  point  forward  through  Part  2,  DoD 
JMASS  will  be  referred  to  as  simply  JMASS. 

Cost  Estimate  Data  Collection.  The  basis  of  the  cost  estimates  came  from  interviews  and  research  with  the 
AF  JMASS  Program  Office,  the  JMASS  Pilot  Projects,  and  the  existing  AF  JMASS  model  development 
community  consisting  of  MSIC  and  NGIC,  both  representatives  of  the  Modeling  and  Simulation  Executive 
Agent  (MSEA)  for  threat.  The  greatest  challenge  in  data  collection  was  finding  legacy  model  cost  data  that 
were  comparable  to  JMASS.  The  legacy  data  were  collected  through  interviews  and  research  within  a 
number  of  acquisition  program  offices.  The  main  challenge  faced  was  the  willingness  of  the  various 
programs  to  share  cost  information  from  their  respective  programs.  Additionally,  the  challenge  in  legacy 
cost  data  collection  was  to  find  legacy  models  that  would  relate  closely  enough  to  JMASS  and  to  make  a 
comparison  between  JMASS  use  and  legacy  use  over  time  in  order  to  assess  the  cost  benefit.  All  of  the  cost 
data  are  measured  in  rough  order  of  magnitude  estimates.  Cost  estimate  data  were  successfully  collected 
and  put  in  context  in  the  following  cost  benefit  analysis. 

Cost  Benefit  Analysis  Foundation.  First,  a  foundation  upon  which  the  cost  benefit  was  based  was 
established.  Since  the  main  concern  was  supporting  weapon  system  acquisition,  the  OSD  T&E  Oversight 
List  which  lists  common  domains  for  the  major  weapon  system  acquisition  programs  in  ACAT  I  and  II 
provided  the  best  starting  point.  Figure  10,  OUSD  DTSEE  T&E  Oversight  List,  displays  the  ACAT  I  and  II 
program  domains  under  T&E  oversight  by  DTSEE.  It  was  logical  to  compile  this  list  into  common  domains 
as  you  see  on  the  left  side  of  the  chart.  It  was  also  logical  to  break  out  the  number  of  programs  in  each 
domain  along  the  time  line  of  acquisition  milestones  and  show  the  distribution  of  programs  across  each 
domain.  The  list  displays  on  the  horizontal  axis  the  quantity  of  programs  in  time  bins  by  milestone.  Each 
milestone  time  bin  contains  the  quantity  of  programs  at  or  earlier  than  the  milestone  listed.  These  common 
domains  form  the  foundation  upon  which  the  analysis  was  built.  The  list  was  compiled  in  this  manner  to 
give  a  picture  of  commonality,  the  potential  to  share  resources,  and  the  sense  of  time  where  numbers  of 
programs  fell  within  the  acquisition  process.  This  break  out  serves  to: 

•  Show  logical  groupings 

•  Provide  a  basis  to  understand  the  common  threads  of  analysis  and  models 

•  Find  a  common  legacy  model  example  used  extensively  throughout  a  domain 

•  Calculate  JMASS  costs  in  viable  areas  that  demonstrate  a  potential  for  reuse 

Best  Domain  Example.  The  focus  of  attention  was  in  the  aircraft  domain  because  this  is  where  the  most 
readily  available  example  was  found.  Figure  11,  Specific  Domain  of  Investigation,  highlights  the  aircraft 
domain  and  the  distribution  of  programs  across  the  domain.  Based  on  research  of  weapons  system 
acquisition  programs  in  this  domain,  the  JSF  program  proved  to  be  the  best  example  for  several  reasons: 
the  program  was  the  most  forthcoming  with  M&S  data;  it  provided  a  viable  common  thread  across  the 
domain  -  commonality  in  aircraft  survivability  analysis  and  modeling;  and  it  provided  the  most  specific 
dollar  cost  estimate  data  on  a  legacy  model  that  is  commonly  used  across  the  whole  aircraft  domain. 
Additionally,  AF  JMASS  model  development  has  focused  in  the  air  domain  with  the  recent  development  of 
environment  models  for  electronic  combat  and  a  number  of  surface-to-air  threat  models.  All  of  these 
conditions  provided  information  needed  to  conduct  the  analysis.  Following  are  the  specifics  of  the  legacy 
modeling  in  the  JSF  program  used  in  the  cost  illustration  examples. 
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Figure  10.  OUSD  DTSEE  T&E  Oversight  List 


ACAT I  and  II  Weapon  Programs  in  Common 


Figure  11.  Specific  Domain  of  Investigation  -  Aircraft 

Legacy  Modeling  in  the  JSF  Program.  The  JSF  program  provided  information  about  M&S  and  analysis 
plans.  Figure  12,  Joint  Strike  Fighter  Legacy  M&S  Suite,  displays  the  models  used  by  JSF  to  address 
analysis  issues.  From  the  J1RD,  the  JMASS  focus  is  engineering/engagement.  The  focus  in  the  study  was 
to  find  a  common  model  supporting  a  common  area  of  analysis  used  throughout  a  common  domain  that  has 
specific  cost  estimate  data  available  to  use  in  the  study  to  compare  with  JMASS.  The  research  with  the  JSF 
M&S  staff  yielded  the  match  of  all  the  conditions  sought.  The  match  within  the  engineering/engagement 
area  allowed  for  the  best  comparison  possible. 
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Figure  12.  Joint  Strike  Fighter  Legacy  M&S  Suite 

Legacy  Common  Thread  -  Enhance  Surface-to-Air  Missile  Simulation  (ESAMS)  .  The  best  common  thread 
found  in  the  engineering/engagement  suite  of  models  was  ESAMS  -  a  SURVIAC  model  used  widely  across 
the  aircraft  domain  by  all  the  Services  to  address  aircraft  survivability  analysis  issues.  Figure  13,  Specific 
Area  of  Investigation  -  ESAMS,  highlights  that  the  model  is  found  within  the  engineering/engagement 
hierarchy  and  addresses  aircraft  survivability  issues.  It  should  be  noted  also  that  the  SURVIAC  program 
office  verified  that  ESAMS  is  used  extensively  across  the  aircraft  domain  by  all  the  Services  for  aircraft 
survivability  analysis.  So,  the  first  stepping-stone  in  the  analysis  process  was  completed  with  the 
identification  of  a  viable  legacy  model  with  which  to  compare  to  JMASS.  The  second  stepping-stone  was 
to  determine  the  cost  estimate  expenditure  of  ESAMS  use  within  the  JSF  program  over  time. 

ESAMS  -  Legacy  Cost  Estimate  Data  from  JSF.  The  JSF  M&S  office  provided  order  of  magnitude  cost 
estimate  data  on  the  use  of  ESAMS  over  the  life  of  the  program.  This  information  was  verified  with  the 
SURVIAC  program  office  which  oversees  the  development,  configuration  control,  use  and  modification  of 
the  model.  The  SURVIAC  office  found  these  estimates  to  be  reasonable.  The  legacy  cost  estimates  of 
ESAMS  will  be  used  in  a  series  of  side-by-side  cost  comparisons  with  JMASS  in  the  sections  to  follow. 
The  assumptions  of  cost  estimate  data  on  JSF  use  of  ESAMS  are: 

•  ESAMS  will  support  analysis  of  JSF  aircraft  survivability  issues  over  a  10  year  period. 

•  The  cost  of  ESAMS  modifications  were  estimated  to  be  between  ~$35K  -  $150K  twice  a  year 
for  each  of  the  10  years.  In  other  words  ~$70K  -  $300K  per  year  over  the  10  year  period  of 
use.  A  low-to-high  range  was  established  because  modifications  vary  due  to  the  complexity  of 
the  analysis  issue. 

JMASS  Cost  Estimate  Data.  This  section  provides  a  detailed  break-down  of  the  estimated  development 
cost  of  AF  JMASS  models.  This  information  was  provided  and  verified  by  the  Air  Force  JMASS  program 
office  and  the  threat  model  developers  at  MSIC.  AF  JMASS  models  are  organized  into  three  areas:  the 
digital  system  model  (DSM)  -  the  representation  of  the  blue  weapon  system;  the  threat  model  -  in  this  case 
the  representation  of  surface-to-air  missile  (SAM)  threat  systems;  and,  the  environment  model.  AF  JMASS 
models  can  be  built  at  varying  levels  of  fidelity  from  a  low  resolution,  “analytic”  model,  to  the  most  robust, 
“EC  capable”,  or  electronic  combat  capable.  Following  is  the  cost  estimate  data  for  JMASS  models: 
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Areas  of  Analysis 
Air-tol4ir 
Vulnerability 
^Survivability 

Lethality 


Specific  Area  of 
Investigation 

(Presentation  of  Cost  Illustration) 


Engineering/Engagement 

Legacy  Models 

RAWLER,  MILAASPEM 
OVART,  FASTGEN 

RADGUNS,  MOSAIC 
SABSEL,  JSEM 


Figure  13.  Specific  Area  of  Investigation  -  ESAMS 

•  Analytic  model  -  cost  estimate  is  ~$400K  per  model  whether  it  is  DSM,  threat,  or 
environment. 

•  EC  Capable  model  -  the  EC  Capable  cost  is  the  cost  of  the  analytic  model  (~$400K)  plus  an 
additional  $600K  to  bring  the  model  to  full  robust  capabilities  for  a  total  of  ~$1M. 

•  Model  component  reuse  -  there  is  a  model  component  reuse  advantage  with  the  analytic 
model.  If  you  are  building  multiple  models  of  a  like  system  you  can  reuse  model  components. 
For  example,  if  you  are  building  five  SAM  threat  systems  at  the  full  EC  capability,  the  first 
model  costs  a  total  of  ~$1M  (~$400K  for  the  analytic  +  ~$600K  to  add  on  the  EC  capability  = 
~$1M)  and  the  other  4  cost  only  ~$600K  each.  This  is  realized  because  the  analytic  piece  of 
the  SAM  system  is  generic  enough  as  a  basis  for  use  in  developing  the  other  four  models.  The 
added  cost  of  ~$600K  for  each  of  the  other  four  SAMS  is  necessary  to  add  on  the  detailed 
unique  characteristics  required  for  an  accurate,  detailed,  fully  robust  representation  of  the 
other  four  SAMS.  Therefore  the  total  cost  to  build  five  EC  capable  SAMs  is  ~$1M  for  the 
first  SAM  +  (~$600K  x  4  other  SAMS)  =  ~$1M  +  ~$2.4M  =  ~$3.4M.  So,  the  total  cost  of 
building  five  EC  capable  SAMS  is  ~$3.4M. 

Cost  Illustrations.  The  next  stepping-stone  in  the  analysis  was  establishing  a  baseline  comparison  between 
the  ESAMS  capability  and  cost  over  time  versus  the  JMASS  capability  and  cost  over  time.  Figure  14,  Cost 
Illustration  Example  -  Side-by-Side  Comparison  (Legacy  vs.  JMASS)  highlights  this  baseline  comparison. 
The  intent  in  making  this  comparison  is  to  make  as  close  to  an  “apples-to-apples”  comparison  as  possible, 
understanding  of  course,  that  there  are  some  differences  between  ESAMS  and  AF  JMASS.  The  two 
systems  are,  for  the  most  part,  comparable  at  the  analytic  level  in  capability.  However,  when  making 
comparisons  at  the  EC  capable  level,  the  models  do  not  compare.  This  is  because  ESAMS  does  not  have 
the  same  highly  detailed  robust  EC  capabilities  as  JMASS.  It  should  be  noted  at  this  point  that  this  is  where 
AF  JMASS  stands  out  as  having  a  greater  benefit  than  the  legacy  system.  As  JMASS  is  an  object-oriented 
system,  it  provides  a  more  disciplined  approach  in  programming.  Although  it  is  possible  the  legacy  could 
be  modified  to  a  similar  EC  capability  it  would  be  cost  prohibitive  to  do  so  because  of  the  extensive  re¬ 
coding  and  verification  in  a  non-object  language. 

Side-By-Side  Cost  Illustration  (Legacy  vs.  JMASS).  Figure  14.  Cost  Illustration,  Side-by-Side  Comparison 
(Legacy  vs.  JMASS)  shows  the  baseline  comparison.  The  JSF  program  is  concerned  with  addressing 
aircraft  survivability  analysis  issues,  using  ESAMS,  to  model  five  different  SAM  threats.  The  side-by-side 
comparison  shows  the  cost  of  ESAMS  compared  to  a  similar  capability  in  JMASS  over  the  same  life  cycle, 
in  rough  order  of  magnitude,  over  10  years. 


DoD  JMASS  Cost  Benefit  Analysis  -  Main  Report 


18 


Criteria  and  Assumptions.  The  criteria  and  assumptions  for  the  comparison  are  listed  at  the  top  of 
Figure  14.  The  criteria  established  for  this  cost  illustration  were  designed  to  focus  the  analysis  into  a 
specific  and  comparable  situation. 

Funding  of  Models.  Funding  of  JMASS  models  is  assumed  to  be  shared  by  the  PM  and  DoD.  The 
PM  would  fund  the  build  of  the  DSM  and  DoD  would  fund  the  build  of  the  threat  and  environment  models; 
All  of  these  models  would  be  placed  in  the  JMASS  modeling  repository  and  made  available  for  reuse. 
Because  JMASS  establishes  a  coding  standard,  model  components  and  models,  all  HLA  compliant,  would 
be  available  for  reuse  to  the  community.  Currently,  PMs  fund  the  modification  of  the  legacy  models. 

Cost  Comparison.  The  left  side  of  the  figure  shows  the  cost  range  of  modifying  the  ESAMS 
model  to  address  the  aircraft  survivability  analysis  issues.  These  cost  estimates  were  introduced  in  the 
previous  section.  The  right  side  of  the  figure  shows  the  comparable  cost  of  building  JMASS  models  to 
address  the  same  analysis  issues.  These  cost  estimates  were  introduced  in  the  previous  section  also.  The 
cost  encompasses  the  build  of  the  three  JMASS  model  types:  the  DSM,  the  threats  -  in  this  case  fiveSAMs, 
and  the  environment.  As  depicted  in  the  figure,  there  is  a  cost  comparison  at  the  analytic  level  and  a  cost 
comparison  at  the  EC  Capable  level.  When  analyzing  each  cost  comparison,  note  that,  at  the  analytic  level, 
the  cost  of  the  three  JMASS  model  types  is  ~$2.8M.  This  cost  falls  within  the  upper  range  cost  of  ESAMS, 
~$0.7M  -  $3.0M.  PMs  are  concerned  with  costs  within  their  programs  and  would  be  interested,  based  on 
the  funding  information  in  the  previous  paragraph,  that  in  the  JMASS  system  they  only  pay  for  the  DSM  -  in 
this  case  ~$0.4M,  a  cost  which  falls  below  the  lower  range  of  the  legacy  cost.  DoD  pays  for  the  threats  and 
environment.  If  a  PM  needs  only  an  analytic  level  capability  to  address  his  analysis  issues,  then  JMASS  is 
a  more  cost  effective  solution.  The  EC  capable  cost  of  the  three  JMASS  model  types  is  ~$5.4M.  This  cost 
is  above  the  upper  range  of  the  legacy  cost.  Given  that  PMs  only  pay  for  the  DSM,  in  this  case  a  cost  of 
~$1.0M,  this  cost  falls  within  the  lower  range  cost  of  ESAMS,  ~$0.7M  -  ~$3.0M. 


Cost  Illustration  (Side-by-Side  Comparison) 

Criteria: 

•  Large  Domain  -  21  A/C 

•  Common  analysis  -A/C  Survivability 

•  Common  model  (ESAMS) 

•  Rough  orders  of  magnitude 

Assumptions: 

•  DoD  JMASS  Cost  -  analytic  ($400K),  EC 
Capable  ($1M) 

•  Threats  and  environments  funded  by  DoD 

•  V&V  Cost  Included 

l.cgacy  MoiL-l  V 

£ ; '  V  ^  C*.  'f?*  lyb’\  C '  Dm  { I  <)\i 
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.  -S0.4M  ~$2.QM 

~$0.4M 

Total  (EC  Capable)  Cost  Prohibitive 

-SI.OM  -S3.4M 

~$  1 ,0M 

PMs  $s  I  PM  $s  DoD  Ss 


Up  Front  Cost  of  DoD  JMASS  is  High  -  Must  be  Available  at  Program  Start 


Figure  14.  Cost  Illustration,  Side-by-Side  Comparison  (Legacy  vs.  JMASS) 

Domain  Reuse  (Four  Programs).  The  next  cost  illustration  will  demonstrate  the  benefit  of  model  reuse  in 
the  JMASS  system.  The  previous  cost  illustration  established  a  foundation  upon  which  this  illustration  will 
be  built.  The  key  points  to  bring  forward  from  the  previous  illustration  are  the  same  basic  model  cost 
estimates  for  ESAMS  and  JMASS  used  in  the  same  context.  The  added  information  in  this  illustration  is 
the  benefit  of  reuse  of  models  already  funded  and  built,  and  available  in  the  model  repository  for  sharing. 
This  cost  illustration  is  framed  using  the  OUSD  T&E  Oversight  List  and  identifying  multiple  programs  in 
the  aircraft  domain  that  show  potential  for  model  sharing  -  reuse.  Figure  15,  Selection  of  Programs  in  a 
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Common  Domain,  highlights  four  programs  in  the  aircraft  domain  early  in  acquisition  that  could  possibly 
share  models.  The  focus  was  narrowed  to  programs  early  in  acquisition,  because  PMs  plan  and  budget  their 
M&S  and  analysis  for  their  program  early.  The  following  cost  illustration  example  demonstrates  the  benefit 
through  reuse  of  threat  and  environment  models  in  a  common  domain. 


OUSD/DTSE&E  T&E  Oversight  List 
ACAT I  and  II  Weapon  Programs  in  Common  Domains 


Program  Domains  from  the 


#  of  Common 
Programs  within 
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Communication*  \ 
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Amphibious  Vehich  p| 
RadarKp 
Space  Vehicle-;-  ' 
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MSI  and  earlier 
programs  will  be 
used  in  the  following 
*  cost  illustration  on 
Domain  Reuse 
(4  programs) 


MS  I  and  earlier 
programs  are 
illustrated  because 
PMs  plan  M&S  use 
early  and  are  not 
likely  to  introduce 
hew  M&S  past  MS  II 


Figure  15.  Selection  of  Programs  in  a  Common  Domain 

Criteria  and  Assumptions .  The  criteria  and  assumptions  for  the  comparison  are  listed  at  the  top  of 
Figure  16,  Cost  Illustration,  Domain  Reuse.  The  criteria  established  for  this  cost  illustration  focused  the 
analysis  into  a  specific  comparable  situation  to  consider  the  benefits  of  reuse.  In  this  illustration  it  should 
be  noted  that  the  threat  and  environment  models  are  built  for  the  B-l  Defense  Systems  Upgrade  Program 
(DSUP)  and  shared  with  the  other  three  programs.  At  the  writing  of  this  report,  the  threat  and  environment 
models  had  already  been  developed  and  delivered  to  B-l  DSUP.  The  B-l  DSUP  program  will  be  the  first 
to  use  AF  JMASS. 

Funding  of  Models .  As  discussed  in  the  previous  cost  illustration,  the  method  of  funding  remains 
the  same  conceptually.  The  difference  in  this  illustration  is  the  Air  Force  has  already  funded  all  the  AF 
JMASS  models  for  B-l  DSUP  and  made  them  available  to  the  model  repository.  The  threat  and 
environment  models  are  assumed  to  be  funded  and  are  available  for  reuse  to  other  programs,  in  this  case,  to 
the  other  three  programs.  The  PMs  would  still  fund  the  build  of  the  their  own  unique  DSM. 

Cost  Comparison .  The  left  side  of  Figure  16  shows  the  cost  range  of  modifying  the  ESAMS 
model  to  address  the  aircraft  survivability  analysis  issues.  The  cost  estimate  figures  were  used  from  the  JSF 
ESAMS  example  as  this  cost  range  is  a  reasonable  rough  order  of  magnitude  estimate  for  all  four  programs. 
These  same  cost  estimates  were  introduced  in  the  previous  section.  Interviews  with  PMs  revealed  that  they 
use  their  own  version  of  ESAMS  and  modify  for  their  specific  need.  This  fact  underscores  the  redundant 
investment  and  duplication  of  effort  made  in  ESAMS  among  these  four  programs.  The  right  side  of  Figure 
16  shows  the  comparable  cost  of  building  JMASS  models  to  address  the  same  analysis  issues.  In  this  case, 
the  threat  and  environment  models  are  already  funded  and  available  and  the  only  cost  to  the  PMs  is  the 
binding  of  their  unique  DSMs.  As  you  see  from  the  Figure  16,  there  is  a  cost  comparison  at  the  analytic 
level  and  a  cost  comparison  at  the  EC  Capable  level.  When  analyzing  each  cost  comparison,  each  of  the 
four  PMs  would  be  interested  to  note,  at  the  analytic  level,  the  cost  of  the  DSMs  is  ~$0.4M.  This  cost  falls 
below  the  low  end  range  of  the  legacy  expenditure.  As  stated  previously,  if  a  PM  needs  only  an  analytic 
level  capability  to  address  analysis  issues,  then  JMASS  is  a  more  cost  effective  solution.  The  EC  capable 
cost  of  the  four  JMASS  DSMs  is  ~$2.8M.  This  cost  falls  exactly  on  the  low  end  cost  range  of  the  collective 
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redundant  investment  made  in  ESAMS  among  the  four  programs.  The  investment  in  JMASS  at  the  EC 
capable  level  would  be  cost  effective  for  these  four  programs  as  the  cost  is  equal  to  the  low  end  legacy  cost. 
JMASS  provides  a  better,  more  robust  capability  at  the  EC  capable  level  and  provides  a  cost  benefit  if 
threat  and  environment  models  are  reusable  within  the  domain. 


Cost  Illustration  (Domain  Reuse) 


Criteria: 

•  Large  Domain  -  21  A/C  (4  pre-MS!) 

•  Common  analysis  -  A/C  Survivability 

•  Common  model  (JSF  ESAMS) 

•  Rough  orders  of  magnitude 


Assumptions: 

•  Threat  and  Environment  models  are 
built  for  B-1  DSUP  funded  by  the  Air 
Force 

•  Four  programs  share  same  threats 
and  environment  built  for  B-1  DSUP 
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Figure  16.  Cost  Illustration,  Domain  Reuse  (Legacy  vs.  JMASS) 

Multiple  Domain  Reuse  (16  Programs).  The  next  cost  illustration  will  demonstrate  the  benefit  of  model 
reuse  of  threat  and  environment  models  across  multiple  domains  sharing  similar  interests.  The  previous 
cost  illustration  established  a  foundation  upon  which  this  illustration  will  follow.  The  key  points  to  bring 
forward  from  the  previous  illustration  are  the  basic  model  cost  estimates  for  ESAMS  and  JMASS  used  in 
the  same  context.  The  added  information  in  this  illustration  is  the  benefit  of  reuse  of  models  among  16 
programs.  This  cost  illustration  is  framed  by  using  the  OUSD  T&E  Oversight  List  and  identifying  multiple 
programs  in  the  aircraft  domain  that  show  potential  for  model  sharing  -  reuse.  Figure  17,  Selection  of 
Programs  in  Multiple  Common  Domain,  highlights  there  are  16  programs  in  the  air  related  domains  early  in 
acquisition  that  could  possibly  share  models.  The  focus  was  narrowed  to  programs  early  in  acquisition, 
because,  based  on  interviews  with  various  PMs,  they  plan  and  budget  the  M&S  and  analysis  for  their 
program  early  and  don’t  typically  initiate  expenditures  later  in  the  program.  The  following  cost  illustration 
example  demonstrates  the  benefit  through  reuse  of  threat  and  environment  models  in  multiple  domains. 

Criteria  and  Assumptions.  The  criteria  and  assumptions  for  the  comparison  are  listed  at  the  top  of 
Figure  18,  Cost  Illustration,  Multiple  Domain  Reuse.  The  criteria  established  for  this  cost  illustration  were 
designed  to  focus  the  analysis  into  a  specific  comparable  situation  and  consider  the  benefits  of  reuse.  In  this 
illustration  it  should  be  noted  that  there  is  a  larger  sample  of  programs  to  support.  It  is  assumed  that  the  16 
programs  will  share  the  same  10  threat  models.  The  10  threat  models  are  different  than  those  were  built  for 
B-1  DSUP  from  the  previous  cost  illustration.  In  this  example,  there  are  new  builds  of  threats  and 
environment  models. 

Funding  of  Models.  It  is  assumed  that  the  funding  of  JMASS  models  would  be  divided  between  the 
PM  and  DoD.  The  PM  would  fund  the  build  of  the  DSM  and  DoD  would  fund  the  build  of  the  threat  and 
environment  models.  All  of  these  models  would  be  placed  in  the  JMASS  modeling  repository  and  made 
available  for  reuse.  Because  JMASS  establishes  a  coding  standard,  model  components  and  models  would 
be  available  for  reuse.  Currently,  PMs  fund  the  modification  of  the  legacy  models. 
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OUSD/DTSE&E  T&E  Oversight  list 
ACAT I  and  II  Weapon  Programs  in  Common  Domains 
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Figure  17.  Selection  of  Programs  in  Multiple  Domains 

Cost  Comparison.  The  left  side  of  the  Figure  18,  shows  the  cost  range  of  modifying  the  ESAMS 
model  to  address  the  aircraft  survivability  analysis  issues.  The  cost  estimate  figures  from  the  JSF  ESAMS 
example  were  used  as  a  reasonable  rough  order  of  magnitude  estimate  for  all  16  programs.  These  same  cost 
estimates  were  introduced  in  the  previous  section.  The  cost  encompasses  the  build  of  the  three  JMASS 
model  types:  the  16  DSMs,  10  threats,  and  one  environment.  As  you  see  from  the  figure  there  is  a  cost 
comparison  at  the  analytic  level  and  a  cost  comparison  at  the  EC  Capable  level.  When  analyzing  each  cost 
comparison  at  the  analytic  level,  the  cost  of  the  three  JMASS  model  types  is  ~$10.8M.  This  cost  falls  below 
the  low  range  cost  of  ESAMS  ~$1 1.2M  -  $48. 0M.  PMs  concerned  with  costs  within  their  programs  would 
be  interested  to  note,  based  on  the  funding  method,  that  in  the  JMASS  system  they  only  pay  for  the  DSM  - 
in  this  case  at  a  lower  cost  than  the  legacy.  DoD  pays  for  the  threats  and  environment.  Again,  if  a  PM 
needs  only  an  analytic  level  capability  to  address  analysis  issues,  JMASS  is  a  more  cost  effective  solution. 
The  EC  capable  cost  of  the  three  JMASS  model  types  is  ~$18.6M.  This  cost  is  within  the  low  end  range  of 
the  legacy  cost.  PMs  only  pay  for  their  DSM.  In  this  case,  cost  of  the  DSMs  is  lower  than  the  legacies 
because  of  the  reuse  of  model  components  and  models.  Intuitively,  with  more  programs  and  sharing,  the 
greater  the  cost  benefit. 

Cost  Benefit.  The  data  represented  in  Figure  19,  Cost  Benefit,  show  the  relationship  of  legacy  costs  to 
DoD  JMASS  costs  from  the  previous  example  of  16  programs.  The  costs  represent  the  total  package  of 
DSM,  threat,  and  environment  models.  The  bar  on  the  top  depicts  the  range  of  cost  of  the  legacy  model 
from  low  to  high  -  the  redundant  investment  of  16  programs  in  the  legacy  system.  The  bar  under  the  legacy 
depicts  the  range  of  cost  of  building  a  suite  of  JMASS  models  (16  DSMs,  10  threats,  and  one 
environment).  The  range  represents  the  low,  the  analytic  level,  to  the  high,  the  EC  capable,  clearly 
depicting  a  long  term  benefit  to  the  department. 


DoD  JMASS  Cost  Benefit  Analysis  -  Main  Report 


22 


Cost  Illustration  (Multiple  Domain  Reuse) 

Criteria:  Assumptions: 

•  Multiple  Domains  -  Aircraft,  Missile,  UAV,  #16  programs  share  same  10  threats  and 

Helicopter  (16  pre-MSI)  one  environment  (different  from  B-1  DSUP) 

•  Common  analysis  -  A/C  Survivability 

•  Common  model  (JSF  ESAMS) 

•  Rough  orders  of  magnitude 
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Helicopter  (2  PMs)  ~$1.4M  -  $  6M 
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Figure  18.  Cost  Illustration,  Multiple  Domain  Reuse  (Legacy  vs.  JMASS) 
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Figure  19.  Cost  Benefit 

Analysis  Summary  Part  2.  The  range  of  JMASS  cost,  analytic  to  EC  is  at  the  low  end  range  of  ROM  cost 
compared  to  legacy.  Based  on  cost  alone,  the  JMASS  analytic  capability  is  cheaper  compared  to  legacy, 
~$10.8M  compared  to  ~$1 1.2M.  To  build  JMASS  at  a  robust  EC  capability,  ~$18.6M,  still  is  at  the  low 
end  range  of  the  legacy  cost  profile  whereas  an  EC  ESAMS  is  cost  prohibitive.  The  PMs  cost  is  lower  when 
they  only  have  to  build  the  DSM.  Although  the  front-end  cost  for  the  system  may  be  high,  JMASS 
demonstrates  the  additional  benefits  gained  above  cost:  reuse  of  model  components  and  models;  elimination 
of  the  duplication  of  legacy  use;  and  an  overall  better  product  with  more  capability  than  what  is  available  in 
the  legacy. 

Assessment  of  Risk.  When  weighing  the  options,  one  must  consider  risk  as  a  factor.  The  analysis  revealed 
several  criteria  that  are  important  to  assessing  risk.  These  criteria  are  presented  with  an  assessment  of  risk 
for  each.  A  scale  is  applied  to  each  criteria  in  the  form  of:  a  minus  sign  (-),  criteria  containing  risk;  or,  in 
the  form  of  a  plus  sign  (+),  criteria  containing  no  risk. 

(-)  User  friendliness.  Based  on  information  from  the  pilot  projects,  the  tools  and  services  of  the 
AF  JMASS  system  still  need  improvement  in  order  to  gain  acceptance  from  the  analysis  community. 

(-)  Timing.  Based  on  information  from  the  pilot  projects,  it  is  estimated  the  time  it  takes  to 
develop  models  and  deliver  them  to  the  model  repository  is  ~2  !4  to  ~5  years.  Over  this  time  period,  the 
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acquisition  programs  identified  at  MSO  and  MSI  will  be  moving  forward  to  MSI  and  MSII  respectively. 
Interviews  with  various  PMs  revealed  they  become  less  receptive  to  new  developments  in  their  programs 
later  in  the  acquisition  cycle. 

(-)  Front-end  Cost.  The  initial  costs  to  build  a  suite  of  JMASS  models  to  support  a  PM  or  group 
of  PMs  within  a  common  domain  is  high.  The  whole  suite  of  JMASS  models  must  be  available  to  the 
program  at  program  start,  the  DSM,  threat  and  environment,  all  capable  of  interoperating  with  each  other. 

(+)  PMs  Cost.  The  PM  cost  is  low,  as  they  now  only  have  to  pay  for  their  DSM. 

(+)  More  disciplined  approach.  JMASS  provides  a  more  disciplined  approach,  in  common  threats 
and  common  environment  models.  These  common  models  provide  a  “level  playing  field”  of  analysis  within 
and  across  multiple  programs. 

(+)  Move  to  Object-Oriented.  An  object-oriented  system  provides  some  significant  benefits: 
improved  model  development  speed;  easier  M&S  reuse;  and  common  look  and  feel  across  simulations 
which  reduces  the  learning  curve  for  executing  and  using  simulations. 

(+)  Reduce  Duplication  of  Effort.  The  object-oriented  system  reduces  the  duplication  of 
simulation  support  elements  and  interfaces:  input/output  services,  time  management,  distributed  interfaces, 
common  support  tools  (e.g.  3-D  viewers,  2-D  plots/strip  charts,  GUIs),  and  common  support  models  (e.g. 
spatial,  math,  terrain). 
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Department  of  Defense 
Joint  Modeling  and  Simulation  System 
Cost  Benefit  Analysis  Report 

Insights 


General.  This  section  is  a  compilation  of  insights  from  the  study,  including  interviews,  categorized  into 
logical  groupings.  The  insights  are  organized  as  follows: 

1.  DoD  JMASS  System 

•  Requirements 

•  Description 

•  Costs 

•  Up  front 

•  Overall 

•  Benefits 

2.  JIRD/JORD 
1.  DoD  JMASS  System 
Requirements 

DoD  JMASS  is  a  notional  acquisition  system  for  the  future,  intended  to  save  time  and  money.  To 
be  efficient  and  effective  technically,  its  models  and  simulations  must  be: 

•  Reusable 

•  Interoperable 

•  Able  to  eliminate  the  legacy  duplication  of  effort 

•  Cost  effective 

•  A  common  model  system  and  model  set 

From  DoD  management’s  perspective,  this  system  cannot  be  so  restrictive  to  the  PM  that  it  stifles 
initiative  or  good  business  practices  (e.g.  PMs  leveraging  Industry  M&S  successes  to  get  the  answers  they 
need)  which  is  a  tenet  of  DoD  Acquisition  Reform.  Top  level  industrial  management  had  to  move  to  M&S 
to  maintain  market  share  and  remain  part  of  the  mainstream  of  manufacturing  technology.  Is  there  a 
requirement  for  a  DoD  M&S  system?  PMs,  consumers  of  analysis,  and  the  analysts  of  the  M&S  community 
who  will  use  it  are  not  asking  for  it,  nor  do  they  have  the  time  or  inclination  to  research  a  new  way  of  doing 
business,  but  the  need  to  find  a  better  way  is  near  term  due  to  the  shrinking  Defense  budget.  Also,  PMs  are 
not  waiting  for  a  system  to  be  built  and  fielded.  They  need  answers,  and  need  them  today,  so  they  are 
answering  their  performance  questions  wherever  they  can  find  the  information.  All  the  Services  share  the 
common  goal  of  producing  a  better,  cheaper  and  faster  -  "sooner  in  the  hands  of  the  warfighter"  -  product. 
The  M&S  system  that  can  satisfy  these  common  goals  within  each  of  the  Services’  specific  warfighter 
domains  will  receive  support. 

AF  JMASS,  as  all  or  part  of  DoD  JMASS,  once  in  its  final  form,  must  demonstrate  its  productivity 
so  that  PMs/analysts  will  want  to  use  it.  Cultural  resistance  will  be  high  to  changing  the  way  DoD  does 
business  because  it  will  be:  1)  New;  2)  No  single  system  has  been  proven  to  fulfill  all  the  requirements  of 
all  the  Services;  and  3)  The  understood  inherent  risk  of  technological  obsolescence  -  hardware  changes 
every  18  months,  software  every  5-6  years,  and  translators  every  6  months. 
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DoD  JMASS  must  hurdle  technical  and  cultural  obstacles,  while  at  the  same  time  minimizing  the 
technological  investment  risk,  to  achieve  DoD  wide  acceptance  as  the  single  acquisition  M&S  system. 

System  Description 

The  system  should  be  a  system  of  systems,  made  up  of  AF  JMASS  and  others  (yet  to  be 
determined).  DoD's  focus  should  be  on  interoperability  and  operational  analysis,  and  Industry  on 
manufacturing  and  design.  AF  JMASS  does  not  fit  all  phases  of  acquisition,  it  only  depicts 
engineering/engagement  models,  and  it  is  not  yet  real  time.  The  acquisition  process  requires  much  more 
than  this,  like  costing,  accounting,  resourcing  and  high  level  analysis.  AF  JMASS  has  the  potential  to 
expand  into  more  areas  of  acquisition  than  just  T&E,  but,  at  this  point  in  time,  it  should  be  viewed  as  one  of 
the  blocks  in  the  foundation  that  will  lead  to  the  fully  robust  DoD  JMASS. 

For  a  full  description  of  DoD  JMASS,  data  format,  threats,  standards,  and  environments  will  need 
to  be  completely  defined  so  that  reuse  and  commonality  will  be  fostered  across  the  process.  The  MSEAs 
that  are  to  oversee  and  maintain  the  environments  will  be  responsible  for  the  libraries  where  the  models  and 
components  will  be  stored  are  far  from  final  resolution.  Legacy  models  don't  easily  transition  to  the  models 
of  the  future,  object-oriented;  in  fact,  based  on  Industry's  experience,  it  is  better  to  just  scrap  legacies  and 
move  to  object-oriented  formats. 

System  Costs 

Up  Front  Costs.  DoD  PMs  want  low  maintenance,  low  investment  vehicles  which  will  save  them 
time  and  money.  The  risk  and  cost  of  new  hardware  and  software  are  things  they  can  ill  afford  and  may  be 
too  high  a  price  to  pay.  Using  Industry  as  an  example,  Chrysler  spent  $  100's  millions  and  Boeing  spent 
around  $2  billion  for  the  move  from  their  old  systems  to  CATIA,  a  single  manufacturing  and  design  M&S 
system.  Leveraging  AF  JMASS  could  save  $30  million,  the  sunk  cost  of  the  existing  system,  and  3  to  5 
years.  Couple  this  with  a  plan  of  DoD  paying  for  the  threat  and  environment  models  and  PMs  paying  for 
the  DSM;  the  M&S  cost  to  the  PM  would  be  drastically  reduced  and  a  standard  would  be  in  place  for 
Industry. 


Overall  Costs.  In  the  long  term,  legacy  models  will  be  replaced  by  some  form  of  object-oriented 
models.  It  is  true  that  the  first  program  that  uses  the  new  models  will  incur  costs  equal  to  or  greater  than 
present  M&S  systems  (rough  order  of  magnitude  for  AF  JMASS  threat  models  -  analytical  $450K,  and  EC 
$1M),  but  the  follow-on  programs  that  can  reuse  the  models  will  get  them  free.  The  Air  Force  has  studied 
1 1  programs  and  estimated  that  the  object-oriented  EC  capable  models  would  cost  $800M,  but  the  reuse  of 
AF  JMASS  model  components  and  models  reduces  this  cost  to  $350M.  The  Air  Force  is  funding  8-12 
analytical  and  5  EC  models,  and  the  environment  model  for  the  B-l  DSUP  program  today. 

System  Benefits 

The  DoD  JMASS  system,  by  fulfilling  its  requirements,  will  support  reuse,  interoperability,  and 
eliminate  the  duplication  of  effort.  It  certainly  has  the  potential  for  substantial  cost  savings  over  time,  while 
increasing  capability  at  costs  equal  to  or  less  than  those  being  paid  for  legacy  models.  The  legacy  models 
used  today  do  not  promote  interoperability.  Programs  tend  to  modify  the  model  version  they  are  using  - 
duplicating  effort,  and  normally  require  "sneaker  net"  operations  to  move  data  from  model  to  model.  The 
capability  of  more  evaluations  completed  with  less  resources  will  certainly  appeal  to  PMs,  but  the 
overarching  embrace  of  object-oriented  technology,  and  a  standard  to  bridge/leverage  Industry's  M&S 
successes  may  be  the  most  appealing  benefit  of  all. 

2.  JIRD/JORD.  The  JIRD,  as  the  only  authoritative  document  for  the  development  of  DoD  JMASS  (the 
JORD  is  still  being  drafted),  defines  DoD  JMASS  as  being  capable  of  doing  many  things  across  the  entire 
domain  of  the  acquisition  process.  These  capabilities  are  too  broad  in  scope  for  an  engineering/engagement 
model  and  simulation  system.  The  pilot  projects,  which  were  designed  to  validate  the  capabilities  of  AF 
JMASS  in  relation  to  the  JIRD,  found  that  object-oriented  is  the  right  approach,  cited  improved  model 
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speed  and  reuse,  and  reduced  duplication  of  effort.  The  Army's  portion  of  the  pilot  projects  was  only  able 
to  prove  16%  of  the  JIRD  requirements  assigned,  yet,  the  version  of  AF  JMASS  being  used  was  an  older 
version  (3.1).  The  latest  version  (3.2),  will  potentially  prove  its  capability  as  it  is  being  employed  by  the  B- 
1  DSUP  program.  Version  (4.0),  available  in  1999,  is  estimated  to  meet  80%  of  the  JIRD  capabilities.  The 
JORD,  as  it  is  finalized,  must  take  into  account  what  AF  JMASS  can  accomplish  and  look  at  a  system  of 
systems  approach  to  build  the  fully  capable  M&S  system  that  the  acquisition  process  requires. 
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Courses  of  Action 


Three  Courses  of  Action.  Upon  review  of  the  background  analysis,  cost  benefit  analysis,  and  insights, 
three  courses  of  action  are  postulated.  The  three  courses  of  action  are  presented  along  with  the  advantages 
and  disadvantages  of  each.  This  information  is  followed  by  a  presentation  of  recommendations  in  the  next 
section. 

•  Course  of  Action  #1.  Invest  in  Air  Force  JMASS. 

Advantages:  It  is  the  most  mature  object-oriented  system.  Investments  have  been  made  in  tools 
and  services  of  system.  Investments  have  been  made  in  EC  Threat  and  Environment  Models  (no 
extra  cost  to  subsequent  users).  The  system  provides  a  more  disciplined  approach  to  analysis  -  use 
of  common  threats  and  environment  models  throughout  the  acquisition  program. 

Disadvantages :  There  is  an  up-front  cost  to  populate  models  for  other  than  EC  work.  The  system 
is  not  yet  widely  used  by  the  analysis  community.  There  is  some  risk  in  the  robustness  of  the  tools 
and  services  of  the  system. 

•  Course  of  Action  #2.  Wait  for  Technology. 

Advantages:  In  the  future  there  may  be  a  possible  single  system  solution,  a  COTS  solution,  which 
may  be  cheaper. 

Disadvantages:  Software  technology  is  continuously  moving  and  improving  -  how  long  do  you 
wait? 

•  Course  of  Action  #3.  Continue  to  use  Legacies. 

Advantages:  No  additional  costs,  analysts  are  comfortable  using  legacy  models. 

Disadvantages:  No  apparent  cost  savings,  continued  duplication  of  effort,  continued  VV&A 
issues  with  the  duplication  of  effort,  lack  of  interoperability. 
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Recommendations 


Invest  in  Air  Force  JMASS.  The  recommendation  is  to  select  Course  of  Action  #1  -  Invest  in  Air  Force 
JMASS.  Make  a  specific  investment  in  air  domains  of  aircraft,  missile,  helicopter,  and  UAV  programs  that 
share  common  interests  by  populating  the  model  repository  with  threat  and  environment  models  common  to 
these  programs.  Make  the  investment  that  meets  the  needs  of  the  Service’s  requirements  in  common 
domains  shared  by  the  Services.  Also,  make  an  investment  into  the  improvements  of  the  tools  and  services 
of  the  system  as  recommended  by  the  pilot  projects  so  that  engineers,  programmers,  and  analysts  will  more 
readily  use  the  system.  Leverage  the  existing  investment  in  threat  and  environment  models  from  the  Bl- 
DSUP  program  to  the  other  air  domains  so  they  will  reap  the  cost  benefit  of  reuse. 

Rationale:  The  Air  Force  has  made,  and  continues  to  make,  investments  in  this  system.  If  DoD 
moved  to  something  other  than  this  system  the  cost  could  be  ~$30M  for  a  new  start.  The  benefit  of 
model  reuse  was  proven  in  the  cost  illustration  examples  -  the  more  expanded  the  domain,  the 
greater  the  benefit. 

Establish  a  Joint  Program  Office  (JPO).  Concurrent  with  the  course  of  action,  it  is  also  recommended 
that  a  Joint  Program  Office  as  a  management  structure  oversee  the  development  of  a  system  that  meets  the 
needs  of  DoD  to  support  acquisition.  There  is  a  need  to  involve  all  stakeholders  in  both  the  acquisition  and 
analysis  communities. 

Rationale:  A  consistent  thread  of  opinions  throughout  the  study  effort  was  the  recommendation 
that  a  JPO  be  formed  to  support  a  tool  structure  for  M&S  in  acquisition. 

Recommend  a  Strategy  for  the  Joint  Program  Office  (JPO).  The  specific  strategy  for  the  Joint  Program 
Office  follows: 

•  Take  an  open-minded  approach,  be  flexible,  keep  options  open  -  a  way  to  do  business. 

•  Aggressively  manage  the  timing  of  implementation  of  the  system. 

•  Monitor  technology  advances  in  object  oriented  systems  -  maintain  sensitivity  to  technological 
obsolescence. 

•  Monitor  the  successes  of  the  B-l  DSUP  use  of  Air  Force  JMASS  -  leverage  successes  into 
other  large  domains  like  munitions. 

•  Investigate  areas  of  common  interest  in  other  DoD  programs  focusing  on  specific  areas  of 
analysis,  sharing  the  benefits  of  the  object  oriented  system. 

•  Take  a  building  block,  phased  approach  to  the  growth  of  the  system  in  other  domains. 

•  Continue  pilot  projects  to  refine  requirements  and  prove  benefits  to  the  stakeholders. 

Rationale:  The  technological  world  is  in  constant  change,  especially  in  computer  related 
technology.  To  avoid  a  situation  of  technological  obsolescence,  it  is  necessary  to  monitor 
carefully  the  ever  changing  software  advances. 

Summary.  The  investment  in  JMASS  is  technology  for  the  future  -  the  object-oriented  approach  meets 
many  needs.  It  supports  a  model  repository  approach  by  fostering  reuse  of  model  components  and  models 
within  and  among  acquisition  programs  that  share  common  interests.  The  move  to  this  system  eliminates 
the  existing  duplication  of  effort  found  among  the  use  of  legacies.  JMASS  provides  a  disciplined  approach 
to  modeling,  provides  interoperability  among  engineering/engagement  models  and  simulations,  and 
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provides  a  standard  for  Industry.  The  JMASS  system  will  initially  address  engineering/engagement  models 
as  described  in  the  JIRD.  The  more  robust  acquisition  needs  will  later  need  to  be  addressed. 
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